Machine Learning for Intangible Cultural Heritage: A Review of Techniques on Dance Analysis

Performing arts and in particular dance is one of the most important domains of Intangible Cultural Heritage. However, preserving, documenting, analyzing and visually understanding choreographic patterns is a challenging task due to technical difficulties it involves. A choreography is a time-varying 3D process (4D) including dynamic co-interactions among different actors (dancers), emotional and style attributes, as well as supplementary ICH elements such as the music tempo, the rhythm, traditional costumes etc. Recent technological advancements have unleashed tremendous possibilities in capturing, documenting and storing Intangible Cultural Heritage content, which can now be generated at a greater volume and quality than ever before. The massive amounts of RGB-D and 3D skeleton data produced by video and motion capture devices. The huge number of different types of existing dances and variations dictate the need for organizing, archiving and analyzing dance-related cultural content in a tractable fashion and with lower computational and storage resource requirements. Motion capturing devices are programmable to extract humans’ skeleton data in terms of 3D points each corresponding to a human joint. This information can be combined with computer graphics software toolkits for modelling, classification and summarization purposes. In this chapter, we present recent trends in choreographic representation in terms of modelling, summarization and choreographic pose recognition. We survey recent approaches employed for the extraction of representative primitives of choreographic sequences, the recognition of choreographic pose and dance movements, as well as for the analysis and semantic representation of choreographic patterns.

舞台芸術、特にダンスは、無形文化遺産の中でも最も重要な領域の一つです。しかし、振り付けのパターンを保存、記録、分析、視覚的に理解することは、技術的な困難さを伴うため、困難な作業です。振り付けは、異なるアクター（ダンサー）間のダイナミックな相互作用、感情やスタイルの属性、さらには音楽のテンポやリズム、伝統的な衣装などの補助的なICH要素を含む、時間的に変化する3Dプロセス（4D）です。近年の技術的進歩により、無形文化遺産の撮影、記録、保存の可能性が大きく広がり、これまで以上に大量かつ高品質なコンテンツが生成されるようになりました。ビデオやモーションキャプチャー機器から生成される膨大な量のRGB-Dおよび3Dスケルトンデータ。膨大な種類の既存のダンスやバリエーションがあるため、ダンスに関連する文化コンテンツを、より簡単に、より少ない計算リソースやストレージリソースで整理し、アーカイブし、分析する必要があります。モーションキャプチャー装置は、人間の関節に対応する3Dポイントという形で人間の骨格データを抽出するようにプログラムされています。この情報は、モデリング、分類、要約を目的としたコンピュータグラフィックスソフトウェアツールキットと組み合わせることができます。本章では、モデリング、サマライズ、振り付けポーズ認識の観点から、振り付け表現の最新動向を紹介する。振り付けシーケンスの代表的なプリミティブの抽出、振り付けポーズやダンスの動きの認識、振り付けパターンの分析と意味的表現など、最近のアプローチを紹介する。

This is a preview of subscription content, log in to check access.

References

Aristidou A, Chrysanthou Y (2013) Motion indexing of different emotional states using LMA components. In: SIGGRAPH Asia 2013 technical briefs, New York, NY, USA, pp 21:1–21:4

Google Scholar

Aristidou P, Stavrakis E, Charalambous P, Chrysanthou Y, Himona SL (2015a) Folk dance evaluation using laban movement analysis. J Comput Cult Herit 8(4):20:1–20:19

Google Scholar

Aristidou A, Charalambous P, Chrysanthou Y (2015b) Emotion analysis and classification: understanding the performers’ emotions using the LMA entities. Comput Graphics Forum 34(6):262–276 (2015)

Google Scholar

Aristidou A, Stavrakis E, Papaefthimiou M, Papagiannakis G, Chrysanthou Y (2018) Style-based motion analysis for dance composition. Vis Comput 34(12):1725–1737

CrossRefGoogle Scholar

Bakalos N, Protopapadakis E, Doulamis A, Doulamis N (2018) Dance posture/steps classification using 3D joints from the kinect sensors. In: 2018 IEEE 16th international conference on dependable, autonomic and secure computing, 16th international conference on pervasive intelligence and computing, 4th international conference on big data intelligence and computing and cyber science and technology congress (DASC/PiCom/DataCom/CyberSciTech), pp 868–873

Google Scholar

Ballas A, Santad T, Sookhanaphibarn K, Choensawat W (2017) Game-based system for learning labanotation using Microsoft Kinect. In: 2017 IEEE 6th global conference on consumer electronics (GCCE), pp 1–3

Google Scholar

Barmpoutis P, Stathaki T, Camarinopoulos S (2019) Skeleton-based human action recognition through third-order tensor representation and spatio-temporal analysis. Inventions 4(1)

Google Scholar

Bouchard D, Badler N (2007) Semantic segmentation of motion capture using laban movement analysis. In: Intelligent virtual agents, pp 37–44

Google Scholar

Chai J, Hodgins JK (2005) Performance animation from low-dimensional control signals. In: ACM SIGGRAPH 2005 papers. New York, NY, USA, pp 686–696

Google Scholar

Chen L, Wei H, Ferryman J (2013) A survey of human motion analysis using depth imagery. Pattern Recogn Lett 34(15):1995–2006

CrossRefGoogle Scholar

Choensawat W, Nakamura M, Hachimura K (2015) GenLaban: a tool for generating labanotation from motion capture data. Multimed Tools Appl 74(23):10823–10846

CrossRefGoogle Scholar

Dewan S, Agarwal S, Singh N (2018) Spatio-temporal laban features for dance style recognition. In: 2018 24th international conference on pattern recognition (ICPR), pp 2911–2916

Google Scholar

Dimitropoulos K et al (2014) Capturing the intangible an introduction to the i-Treasures project. In: 2014 international conference on computer vision theory and applications (VISAPP), vol 2, pp 773–781

Google Scholar

Doulamis A et al (2013) 4D reconstruction of the past. In: First international conference on remote sensing and geoinformation of the environment (RSCy2013), vol 8795, p 87950

Google Scholar

Doulamis N, Doulamis A, Ioannidis C, Klein M, Ioannides M (2017) Modelling of static and moving objects: digitizing tangible and intangible cultural heritage. In: Mixed reality and gamification for cultural heritage, Springer, Cham, pp 567–589

Google Scholar

Elhamifar E, Sapiro G, Vidal R (2012) See all by looking at a few: sparse modeling for finding representative objects. In: 2012 IEEE conference on computer vision and pattern recognition, pp 1600–1607

Google Scholar

Ferguson S, Schubert E, Stevens CJ (2014) Dynamic dance warping: using dynamic time warping to compare dance movement performed under different conditions. In: Proceedings of the 2014 international workshop on movement and computing, New York, NY, USA, pp 94:94

Google Scholar

Hachimura K, Takashina K, Yoshimura M (2005) Analysis and evaluation of dancing movement based on LMA. In: ROMAN 2005. IEEE international workshop on robot and human interactive communication, pp 294–299

Google Scholar

Hajdin M, Kico I, Dolezal M, Chmelik J, Doulamis A, Liarokapis F (2019) Digitization and visualization of movements of slovak folk dances. In: The challenges of the digital transformation in education, pp 245–256

Google Scholar

Hisatomi K, Katayama M, Tomiyama K, Iwadate Y (2011) 3D archive system for traditional performing arts. Int J Comput Vis 94(1):78–88

CrossRefGoogle Scholar

Hutchinson A, Hutchinson WA, Guest AH (1970) Labanotation: or, kinetography Laban: the system of analyzing and recording movement. Taylor & Francis

Google Scholar

Isola P, Zhu J-Y, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), Honolulu, HI, pp 5967–5976

Google Scholar

Kahol K, Tripathi P, Panchanathan S (2004) Automated gesture segmentation from dance sequences. In: Proceedings of 2004 sixth IEEE international conference on automatic face and gesture recognition, pp 883–888

Google Scholar

Kapadia M, Chiang I, Thomas T, Badler NI, Kider JT Jr (2013) Efficient motion retrieval in large motion databases. In: Proceedings of the ACM SIGGRAPH symposium on interactive 3D graphics and games, New York, NY, USA, pp 19–28

Google Scholar

Kavakli E, Bakogianni S, Damianakis A, Loumou M, Tsatsos D (2004) Traditional dance and E-learning: the WebDance learning environment

Google Scholar

Kavouras I, Protopapadakis E, Doulamis A, Doulamis N (2019) Skeleton extraction of dance sequences from 3D points using convolutional neural networks based on a new developed C3D visualization interface. In: The challenges of the digital transformation in education, pp 267–279

Google Scholar

Kico I, Grammalidis N, Christidis Y, Liarokapis F (2018) Digitization and visualization of folk dances in cultural heritage: a review. Inventions 3(4):72

CrossRefGoogle Scholar

Kim D, Jang M, Yoon Y, Kim J (2015) Classification of dance motions with depth cameras using subsequence dynamic time warping. In: 2015 8th international conference on signal processing, image processing and pattern recognition (SIP), pp 5–8

Google Scholar

Kitsikidis A et al (2015) A game-like application for dance learning using a natural human computer interface. In: Universal access in human-computer interaction. Access to Learning, Health and Well-Being, pp 472–482

Google Scholar

Kitsikidis A, Dimitropoulos K, Douka S, Grammalidis N (2018) Dance analysis using multiple Kinect sensors. In: 2014 international conference on computer vision theory and applications (VISAPP), vol 2, pp 789–795

Google Scholar

Kohn B, Nowakowska A, Belbachir AN (2012) Real-time body motion analysis for dance pattern recognition. In: 2012 IEEE computer society conference on computer vision and pattern recognition workshops, pp 48–53

Google Scholar

Kojima K, Hachimura K, Nakamura M (2009) LabanEditor: graphical editor for dance notation. In: 11th IEEE international workshop on robot and human interactive communication proceedings, pp 59–64

Google Scholar

Krüger B, Tautges J, Weber A, Zinke A (2010) Fast local and global similarity searches in large motion capture databases. In: Proceedings of the 2010 ACM SIGGRAPH/eurographics symposium on computer animation, Goslar Germany, Germany, pp 1–10

Google Scholar

Krüger B, Vögele A, Willig T, Yao A, Klein R, Weber A (2017) Efficient unsupervised temporal segmentation of motion data. IEEE Trans Multimed 19(4):797–812

CrossRefGoogle Scholar

Lee J, Chai J, Reitsma PSA, Hodgins JK, Pollard NS (2002) Interactive control of avatars animated with human motion data. In: Proceedings of the 29th annual conference on computer graphics and interactive techniques, New York, NY, USA, pp 491–500

Google Scholar

Liutkus A, Dremeau A, Alexiadis D, Essid S, Daras P (2012) Analysis of dance movements using gaussian processes: extended abstract. In: Proceedings of the 20th ACM international conference on multimedia, New York, NY, USA, pp 1375–1376

Google Scholar

Masurelle A, Essid S, Richard G (2013) Multimodal classification of dance movements using body joint trajectories and step sounds. In: 2013 14th international workshop on image analysis for multimedia interactive services (WIAMIS), pp 1–4

Google Scholar

Mehta D et al (2017) Monocular 3D human pose estimation in the wild using improved CNN supervision. In: 2017 international conference on 3D vision (3DV), pp 506–516

Google Scholar

Protopapadakis E, Grammatikopoulou A, Doulamis A, Grammalidis N (2017) Folk dance pattern recognition over depth images acquired via kinect sensor. ISPRS-Int Arch Photogramm Remote Sens Spat Inf Sci 587–593

Google Scholar

Protopapadakis E, Voulodimos A, Doulamis A, Camarinopoulos S, Doulamis N, Miaoulis G (2018) Dance pose identification from motion capture data: a comparison of classifiers. Technologies 6(1):31

CrossRefGoogle Scholar

Rallis I, Georgoulas I, Doulamis N, Voulodimos A, Terzopoulos P (2017) Extraction of key postures from 3D human motion data for choreography summarization. In: 2017 9th international conference on virtual worlds and games for serious applications (VS-Games), pp 94–101

Google Scholar

Rallis I, Langis A, Georgoulas I, Voulodimos A, Doulamis N, Doulamis A (2018a) An embodied learning game using kinect and labanotation for analysis and visualization of dance kinesiology. In: 2018 10th international conference on virtual worlds and games for serious applications (VS-Games), pp 1–8

Google Scholar

Rallis I, Doulamis N, Doulamis A, Voulodimos A, Vescoukis V (2018b) Spatio-temporal summarization of dance choreographies. Comput Graph 73:88–101

Google Scholar

Raptis M, Kirovski D, Hoppe H (2011) Real-time classification of dance gestures from skeleton animation. In: Proceedings of the 2011 ACM SIGGRAPH/eurographics symposium on computer animation, New York, NY, USA, pp 147–156

Google Scholar

Rizzo A et al (2018) WhoLoDancE: whole-body interaction learning for dance education

Google Scholar

Shay A, Sellers-Young B (2016) The Oxford handbook of dance and ethnicity. Oxford University Press

Google Scholar

Shiratori T, Nakazawa A, Ikeuchi K (2006) Dancing-to-music character animation. Comput Graph Forum 25(3):449–458

CrossRefGoogle Scholar

Tang T, Jia J, Mao H (2018) Dance with melody: an LSTM-autoencoder approach to music-oriented dance synthesis. In: Proceedings of the 26th ACM international conference on multimedia, New York, NY, USA, pp 1598–1606

Google Scholar

Voulodimos A, Doulamis N, Fritsch D, Makantasis K, Doulamis A, Klein M (2016) Four-dimensional reconstruction of cultural heritage sites based on photogrammetry and clustering. J Electron Imaging 26:011013

CrossRefGoogle Scholar

Voulodimos A, Rallis I, Doulamis N (2018a) Physics-based keyframe selection for human motion summarization. Multimed Tools Appl

Google Scholar

Voulodimos A, Doulamis N, Doulamis A, Rallis I (2018b) Kinematics-based extraction of salient 3D human motion data for summarization of choreographic sequences. In: 2018 24th international conference on pattern recognition (ICPR), pp 3013–3018

Google Scholar

Wang J, Miao Z, Guo H, Zhou Z, Wu H (2017) Using automatic generation of Labanotation to protect folk dance. JEI 26(1):011028

Google Scholar

Wang T-C, Liu M-Y, Zhu J-Y, Tao A, Kautz J, Catanzaro B (2018) High-resolution image synthesis and semantic manipulation with conditional GANs. In: 2018 IEEE/CVF conference on computer vision and pattern recognition, Salt Lake City, UT, USA, pp 8798–8807

Google Scholar

Wolf W (1996) Key frame selection by motion analysis. In: 1996 IEEE international conference on acoustics, speech, and signal processing conference proceedings, vol 2, pp 1228–1231

Google Scholar

Zacharatos H, Gatzoulis C, Chrysanthou Y, Aristidou A (2013) Emotion recognition for exergames using laban movement analysis. In: Proceedings of motion on games, New York, NY, USA, pp 39:61–39:66

Google Scholar

Zhang Z (2012) Microsoft kinect sensor and its effect. IEEE Multimed 19(2):4–10

CrossRefGoogle Scholar

Zhou F, Torre FD, Hodgins JK (2013) Hierarchical aligned cluster analysis for temporal clustering of human motion. IEEE Trans Pattern Anal Mach Intell 35(3):582–596

Google Scholar